CLDSRV-873: return checksum in HeadObject and GetObject by leif-scality · Pull Request #6117 · scality/cloudserver

leif-scality · 2026-03-18T14:19:37Z

Return checksum in GetObject
Return checksum in HeadObject
Skip returning checksums for parts for the moment, part checksum not yet stored https://scality.atlassian.net/browse/S3C-11073

lib/api/objectHead.js

claude · 2026-03-18T14:22:10Z

PR title mentions GetObject but only HeadObject has checksum support added. lib/api/objectGet.js is unchanged. Either implement GetObject checksum mode or correct the title.

Review by Claude Code

codecov · 2026-03-18T14:23:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.73%. Comparing base (8f9c848) to head (202d8e0).
⚠️ Report is 5 commits behind head on development/9.4.
✅ All tests successful. No failed tests found.

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/api/objectGet.js	`84.74% <100.00%> (+0.81%)`	⬆️
lib/api/objectHead.js	`91.57% <100.00%> (+0.88%)`	⬆️

... and 1 file with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.4    #6117      +/-   ##
===================================================
- Coverage            84.76%   84.73%   -0.04%     
===================================================
  Files                  207      207              
  Lines                13569    13587      +18     
===================================================
+ Hits                 11502    11513      +11     
- Misses                2067     2074       +7

Flag	Coverage Δ
file-ft-tests	`68.64% <100.00%> (+0.04%)`	⬆️
kmip-ft-tests	`28.07% <0.00%> (-0.04%)`	⬇️
mongo-v0-ft-tests	`69.83% <100.00%> (+0.01%)`	⬆️
mongo-v1-ft-tests	`69.82% <100.00%> (-0.04%)`	⬇️
multiple-backend	`36.48% <60.00%> (+0.02%)`	⬆️
sur-tests	`35.53% <0.00%> (-0.08%)`	⬇️
sur-tests-inflights	`37.35% <0.00%> (-0.07%)`	⬇️
unit	`71.03% <100.00%> (+0.36%)`	⬆️
utapi-v2-tests	`34.48% <40.00%> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR adds end-to-end checksum handling: computing/validating checksums on incoming PUT/UploadPart streams, persisting them in object metadata, and returning checksum headers in responses (notably for HeadObject when checksum mode is enabled).

Changes:

Add checksum parsing/validation utilities and a new ChecksumTransform stream to compute digests while streaming uploads.
Store computed checksum data into metadata (including defaulting to CRC64NVME when no checksum is provided) and return checksum response headers for PutObject and HeadObject (x-amz-checksum-mode: ENABLED).
Expand unit/functional test coverage for checksum headers, trailer parsing, and rejection paths.

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
yarn.lock	Pins `arsenal` to a specific commit on the checksum-metadata branch.
package.json	Updates `arsenal` dependency to a git branch providing checksum metadata support.
lib/services.js	Persists checksum into `ObjectMD` and returns checksum in metadataStoreObject result.
lib/routes/routeBackbeat.js	Explicitly ignores computed checksum from `dataStore` for Backbeat data-only writes.
lib/auth/streamingV4/trailingChecksumTransform.js	Parses trailing checksum trailer and emits a `trailer` event for downstream validation.
lib/auth/streamingV4/ChecksumTransform.js	New transform to compute per-algorithm digests and validate expected/trailer checksums.
lib/api/objectPut.js	Returns `x-amz-checksum-*` response header when checksum is available from storage result.
lib/api/objectHead.js	Adds `x-amz-checksum-mode` validation and returns checksum headers when enabled.
lib/api/apiUtils/object/storeObject.js	Integrates prepareStream checksum pipeline, validates checksum post-put, and deletes data on checksum failure.
lib/api/apiUtils/object/prepareStream.js	Builds stream pipeline (V4 / trailing trailer parsing / checksum transform) and returns `{ error, stream }`.
lib/api/apiUtils/object/createAndStoreObject.js	Adds checksum handling for zero-byte objects (computes and stores empty-body checksum).
lib/api/apiUtils/integrity/validateChecksums.js	Adds trailer-related checksum error types, header parsing (`getChecksumDataFromHeaders`), and error mapping.
tests/unit/auth/TrailingChecksumTransform.js	Refactors and expands trailer parsing tests; updates expectations for new error behavior.
tests/unit/auth/ChecksumTransform.js	New unit tests for checksum computation and trailer/non-trailer validation logic.
tests/unit/api/utils/metadataMockColdStorage.js	Exports `baseMd` for reuse in checksum-mode unit tests.
tests/unit/api/objectPut.js	Adds unit tests asserting checksum stored in metadata and returned in response headers.
tests/unit/api/objectHead.js	Adds unit tests for checksum-mode response headers across algorithms.
tests/unit/api/apiUtils/object/storeObject.js	New tests for `dataStore()` checksum validation/cleanup and callback correctness.
tests/unit/api/apiUtils/object/prepareStream.js	New tests validating prepareStream return shape and pipeline behavior for multiple modes.
tests/unit/api/apiUtils/integrity/validateChecksums.js	Adds unit tests for new header parsing and error mapping helpers.
tests/multipleBackend/routes/routeBackbeat.js	Adds checksum validation expectations for Backbeat data write routes.
tests/functional/raw-node/test/trailingChecksums.js	Updates trailing checksum digest constant in raw-node functional test.
tests/functional/raw-node/test/routes/routeMetadata.js	Adds end-to-end verification that checksum is stored in metadata after PutObject.
tests/functional/raw-node/test/checksumPutObjectUploadPart.js	New extensive raw-node functional suite covering checksum rejection and trailer scenarios.
tests/functional/aws-node-sdk/test/object/putVersion.js	Updates metadata comparison exclusions to include `checksum`.
tests/functional/aws-node-sdk/test/object/objectHead.js	Adds SDK-level HeadObject checksum-mode tests and CRT CRC64NVME wiring.
tests/functional/aws-node-sdk/test/object/mpuVersion.js	Adds workaround for MPU restore path that does not yet persist checksums.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lib/api/objectHead.js

package.json

lib/auth/streamingV4/ChecksumTransform.js

lib/api/apiUtils/integrity/validateChecksums.js

bert-e · 2026-03-27T15:38:49Z

Hello leif-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-27T15:38:53Z

Branches have diverged

This pull request's source branch improvement/CLDSRV-873-head-get-object-return-checksum has diverged from
development/9.4 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/9.4 into improvement/CLDSRV-873-head-get-object-return-checksum
Rebase improvement/CLDSRV-873-head-get-object-return-checksum onto origin/development/9.4

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2026-03-27T17:05:15Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-873 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.4.0

Please check the Fix Version/s of CLDSRV-873, or the target
branch of this pull request.

lib/api/objectHead.js

claude · 2026-03-27T17:09:20Z

PR title says "HeadObject and GetObject" but only HeadObject (objectHead.js) is modified — GetObject changes appear to be missing
- Add the same checksum logic to objectGet.js, or update the PR title if GetObject is in a separate PR
Nit: header name interpolates checksumAlgorithm directly without .toLowerCase() — currently safe because the write path stores lowercase, but fragile if that ever changes
- Add .toLowerCase() to the interpolation for defensive safety
  
  Review by Claude Code

lib/api/objectHead.js

claude · 2026-03-30T14:10:07Z

PR title says "HeadObject and GetObject" but only HeadObject is implemented — objectGet.js has no checksum-mode handling
- Either add GetObject support or update the title to reflect the actual scope

Review by Claude Code

lib/api/objectHead.js

claude · 2026-03-30T14:33:37Z

PR title claims changes to both HeadObject and GetObject, but objectGet.js has no changes. Either the title is wrong or GetObject checksum support is missing from this PR.
- Update PR title to reflect only HeadObject, or add the GetObject implementation.
- Minor: consider adding .toLowerCase() to checksum.checksumAlgorithm when building the response header name, as a defensive measure against metadata inconsistencies.

Review by Claude Code

lib/api/objectHead.js

tests/functional/aws-node-sdk/test/object/objectHead.js

claude · 2026-03-30T15:48:49Z

HeadObject returns full-object checksum even for partNumber requests, but GetObject correctly skips it. The guard in objectHead.js (line 106) should also exclude partNumber to avoid returning incorrect checksums (same rationale as S3C-11073).
- Add a partNumber check: if (checksumMode === 'ENABLED' && partNumber === undefined)
- In the HEAD functional test, checksumAlgorithms is declared after it is used in the before() hook. Move it before the before() hook for clarity.
- Match the declaration order used in the GET functional test

Review by Claude Code

claude · 2026-03-30T15:58:00Z

LGTM

Review by Claude Code

claude · 2026-03-30T16:18:01Z

LGTM

Review by Claude Code

claude · 2026-03-31T13:12:30Z

LGTM

Review by Claude Code

leif-scality · 2026-03-31T13:35:46Z

ping

bert-e · 2026-03-31T13:35:57Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

leif-scality · 2026-03-31T13:36:10Z

/approve

bert-e · 2026-03-31T13:36:23Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2
development/9.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2026-03-31T14:01:47Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.4

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2
development/9.3

Please check the status of the associated issue CLDSRV-873.

Goodbye leif-scality.

leif-scality changed the title ~~CLDSRV-873: return checksum in HeadObject~~ CLDSRV-873: return checksum in HeadObject and GetObject Mar 18, 2026

claude bot reviewed Mar 18, 2026

View reviewed changes

lib/api/objectHead.js Outdated Show resolved Hide resolved

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch 2 times, most recently from 7d130cc to 8e51413 Compare March 20, 2026 15:49

nicolas2bert requested a review from Copilot March 25, 2026 18:30

Copilot started reviewing on behalf of nicolas2bert March 25, 2026 18:31 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch 3 times, most recently from fea5f58 to 8f9c848 Compare March 27, 2026 15:11

Base automatically changed from improvement/CLDSRV-872-store-checksum-in-object-metadata to development/9.4 March 27, 2026 15:38

leif-scality closed this Mar 27, 2026

leif-scality force-pushed the improvement/CLDSRV-873-head-get-object-return-checksum branch from 8626736 to 8f9c848 Compare March 27, 2026 17:04

leif-scality reopened this Mar 27, 2026

claude bot reviewed Mar 27, 2026

View reviewed changes

lib/api/objectHead.js Outdated Show resolved Hide resolved

claude bot reviewed Mar 27, 2026

View reviewed changes

lib/api/objectHead.js Show resolved Hide resolved

leif-scality force-pushed the improvement/CLDSRV-873-head-get-object-return-checksum branch from 585ee15 to 6456d26 Compare March 30, 2026 14:07